The proliferation of digital image and video capture devices has led to multiple modalities of capture being present at any picture-taking occasion. For example, it is possible to have both videos and still images since most digital cameras now support capture of video clips; and digital camcorders can capture still images. In an important family event or a public event, such as weddings and sports matches, there are usually multiple still and video capture devices capturing the scene simultaneously. This scenario results in videos and stills that overlap in time. For instance, multiple stills may be captured during the duration of a video clip and multiple video sequences may overlap to various degrees. The current state of the art in consumer image management software, such as Google Picasa, Adobe Photo Album and Kodak EasyShare, display still images and videos in chronological order with no ability to indicate overlapping captures. In some cases, the date/time of file creation (not capture date/time) is used for video, which effectively removes video clips from the natural timeline and places them at one end of a batch of media transferred from capture device to storage device. In the best cases, videos are inserted at the point in the timeline indicated by the start of capture. Still images or video captured during the duration of a longer video clip appear after its thumbnail representation, with no indication of possible overlap; where overlaps could be in time or another relevant concept such as location or event.
This mode of display makes it difficult to pick the best representation of a given moment; choose between different modalities or create composites of different modalities. An alternative is to provide browsing mechanisms that explicitly show overlaps between captures of one or more modalities and also allows the user to switch between them on a UI display.
In U.S. Pat. No. 6,950,989, Rosenzweig et al describe a timeline-based browsing view for image collections. The images in the collection can be viewed at different time granularity (year-by-year, month-by-month etc), and also along other metadata such as location taken and people in picture. However, it is assumed that all media in the collection can be placed in order on the timeline, and overlaps in time between media are not handled.
A few patents discuss some aspects of media overlaps in time or media captured at the same event, but in very limited circumstances, and in contexts other than browsing a consumer image collection. In U.S. Pat. No. 6,701,014, Syeda-Mahmood describes a way to associate slides (say in Microsoft PowerPoint) to slides that are being shown on a screen in a video of the presentation. In U.S. Pat. No. 7,102,644, Hoddie et al describe a way to embed movies within a movie, in cases where there is overlap in content between them. The intention is to allow video editors to edit all the related clips at the same time, so that any changes made in one stream can be reflected in the other related ones. In U.S. Pat. No. 7,028,264, Santoro et al describe an interface that shows multiple sources on the same screen, but these sources are not related to each other and are not linked in any way. For example, the sources could be different television channels covering the news, sports, weather and stocks. In U.S. Pat. No. 6,978,047, Montgomery describes storing multiple views of the same event for surveillance applications, but in this case, the video cameras are synchronized. This system does not provide means for relating asynchronous captures that occur in the consumer event captures, and there is no browsing interface provided. In U.S. Pat. No. 7,158,689, Valleriano et al handle asynchronously captured images of an event, but the event type is a special case of a timed event such as a race, and contestants are tracked at various fixed stations. These methods are specific to the applications being described, and provides no framework for handling the generalized problem of browsing multiple sources of media captured asynchronously at the same event.